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Introduction 

In 1976, the National Science Foundation called for an investigation of the status 

of science education. Subsequently, Weiss (1978) reported that the school science could 

be described vsdth one word, textbook. Textbooks determined the content of instruction 

as well as teaching procedures in thousands of classrooms. Such a statement may not be 

accord with the usual theory in education, but it was reported by schools’ supervisors and 

states’ inspectors of instructional materials (Harms & Yager, 1981) year after year. 

Recently, the Third International Mathematics and Science Study (TIMSS) (Schmidt, 

McKnight, & Raizen, 1 996) indicates that teachers throughout the world base about 50 

percent of their weekly teaching time on textbooks. Textbooks persistently have had 

great influence on what is taught and how it is delivered in science. 

Few individuals deny the important roles of science teachers in transforming 

cmrent science education. However, while science teacher seems to use textbooks most 

of the time, it is curious that so little effort has been devoted to analyzing cmriculum 

materials (Good, 1993); despite this urgent need. Among six Science Teaching Standards 

stated in the National Science Education Standards (National Research Council, 1 996), 

one sub-standard of Teaching Standard D describes explicitly that 

Effective science teaching depends on the availability and organization of 
materials, equipment, media, and technology . . . Teachers must be given the 
resources and authority to select the most appropriate materials and to make 
decisions about when, where, and how to make them accessible. (NRC, 1996, p. 
44-45) 

However, without an effective and efficient approach to evaluating instructional 
materials, teacher preparation will always fail to provide the professional training in 
selecting, analyzing, and utilizing proper curriculum materials. Thus, deciding which 
materials, media and even textbooks to assist teaching becomes another uncomfortable 
chore required of teachers. 

Research devoted to the analysis of texts has been conducted periodically 
throughout the history of education. The common methodology applied by researchers to 
examine science textbooks has been a form of content analysis. However, the content 
analysis method had no standard set of rules or guidelines. In general, textbooks studies 
claiming to apply content analysis were profoundly influenced by Berelson’s (1952) 
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ideology. To be objective and systematic, these studies usually excluded qualitative 
considerations of what “content” and instead simply counted qualities of the 
representative “content.” A central idea in content analysis is classifying words of a text 
into a smaller number of content categories (Weber, 1985). By contrast, users of 
textbooks do not use textbooks in that quantitative fashion. To effective learning, the 
quality of the content is crucial, not the quantity of words in a content category. 

To develop a more scientific and systematic approach for analyzing the quality of 
science textbooks, Krippendorff s (1980) approach to content analysis may be one 
solution. Krippendorff defined content analysis as a research technique for making 
replicable and valid inferences from data to their context. The means to a valid and 
replicable analysis depends on a clear set of frameworks, recognized as the “conceptual 
foundations” to guide the investigation. Accordingly, the past science textbooks studies 
conducted between 1989 to 1996 were generally based on two approaches to establish 
conceptual frameworks. The first approach is that a framework with theoretical support 
prior to content analysis will be generated, such as the “scientific literacy” framework 
generated by Chiappetta, Fillman, and Sethna (1991), and later applied by Lumpe & Beck 
(1996). The second approach is that the conceptual framework is explored, constructed, 
and refined during process of content analysis, such as the approach applied by Jeffery & 
Roach (1994) in their study of evolution protoconcepts. A question emerges from these 
two approaches, which approach is closer to teachers’ approach when textbook-related 
decisions needed to be made? 

In addition to conceptual framework matter, another issue emerges from the 
content analysis studies of science textbooks is the amount of texts for analysis. One 
trend is randomly selection of certain portions of the textbook, and another is whole 
textbook examination for a specific content. Garcia (1985) found randomly selection of 
5% pages of each textbook being sufficient to represent textbook content. Hershey 
(1996) warns that the technique of randomly selecting science textbook pages for themes 
analysis may “easily skew the results because some chapters emphasize [one theme] 
much more than the other [themes]” (p.328). Yet, whenever teachers have to select a 
textbook, how much of the textbook was actually examined? 
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Purpose of the Study 

This study is devoted to design a content analysis method that is real to daily 
instructional practice. Two guiding research questions will be answered before the 
method being proposed. 1) What are the trends for conducting textbook content analysis 
method in the science educational research? 2) To what extend do these trends address 
daily instructional needs? Specifically, this study wishes to recognize that what are the 
needs for science teacher educators to prepare qualified science teachers in terms of 
readiness in their capability of examining instructional materials like textbooks. 

The structure of this paper is arranged as: review of literature on the role of 
instructional materials, especially textbooks in science education research; review of 
literature on past content analysis method for science textbooks; review of research of 
instructional needs and recommendations; conclusion and a proposal of a content analysis 
method for teacher education. 

Roles of Textbooks in Science Education 

Unlike the standards documents that seek to guide curriculum, instructional 
materials are concrete to practitioners, who use them in their daily routine. Instructional 
materials are what teachers and students use in their classrooms. Science textbooks have 
been known for decades as a dominant instructional tool in science education. It is the 
textbook in thousands of classrooms that determine the content of instruction as well as 
the teaching procedures. 

This view may not be in accord with the usual theory of education, but it is 
supported by the facts as reported by supervisors and state inspectors of schools for the 
past few decades. Recently, this phenomenon was again evidenced by the Third 
International Mathematics and Science Study (TIMSS) (lEA, 1996), in which it is shown 
that teachers throughout the world use textbooks to guide their science instruction, with 
science teachers basing about 50 percent of their weekly teaching time on textbooks. 
Thus, science textbooks have great influence over how content is delivered and even 
what should be taught. 
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The TIMSS report also shows that the long-time admonition of “Less is More!” 
has rarely been followed. Even though the core science topics are similar across the 50 
states, nevertheless, with state administrators selecting topics based on their visions of 
education, only a few topics are common to all the States. Thus, the highly competitive 
textbook market has caused publishers to include as many topics as they can, and this has 
resulted in the thickest textbooks found in the world — however shallow they may be in 
content elaboration. The nature of the instructional strategies for teaching and learning a 
specific topic must be elaborated to provide teachers with appropriate idea about how to 
explore specific concepts with their students to reinforce learning. Addressing the 
problems associated with instructional materials is an important function of education 
reform. 

Recognizing the role of science textbooks in instruction does not intend to 
encourage teaching by textbooks but to bring teacher educators’ attention to such a need 
for teacher preparation. Such need was recognized by deBerg (1989), who described the 
difficulty science teachers have in gaining access to primary source materials relating to 
science. Thus, science teachers may turn to science textbooks, which are known for their 
shortage of authenticity. In addition to content issue, deBerg (1989) chose to examine the 
history of science in textbooks. He noted the lack of development of an instructional 
sense in the history of science content in textbooks. According to deBerg, the use of 
exercises, projects, assignments, debates, and discussions by students, which relate to the 
textbook’s history of science content, are absent from all textbooks examined. It is 
possible that since the history of science is virtually nonexistent in the instructional sense, 
students are getting very little exposure to understand the purpose of the historical 
elements inclusion. 

Ball and Cohen’s (1996) idea of providing teachers with: a) comfort and 
confidence in content, and b) the pedagogy of the content; has not happened in teacher 
education yet. Such preparation is perceived as one indispensable component of 
teachers’ willingness to change their instructional paradigm for innovative teaching. 
“Teachers are central to education, but they must not be placed in the position of being 
solely responsible for reform. Teachers will need to work within a collegial, 
organizational, and policy context that is supportive of good science teaching” (NRC, 
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1996, p. 27). Standards-based science education reform will need more than the 
documentation of science standards reports; there is a pressing need to investigate the 
readiness of science teachers to approach quality science teaching. As one criteria of 
quality teaching endorsed in standards document as indicator of wisely utilizing 
instructional materials, teacher education will need to address such emerging need. 

Research of Content Analysis Methodology 

Content analysis is a research technique for making replicable and valid 

inferences from data to their context. (Krippendorff, 1980, p. 21) 

To analyze textbook’s content, a set of criteria or frameworks, recognized as the 
“conceptual foundations” that guide the investigation, is a must. Conceptual frameworks 
portray “prescriptive, analytical, and methodological purposes,” which can avoid cursory 
and subjective analysis. The objective of this section is to review content analysis 
techniques commonly applied by researchers by scrutinizing the 3 1 studies of science 
textbook content analysis conducted during the years of 1989 to 1996. 

Baker (1991), Finley, Lawrenz, and Heller (1992), and Lederman, Gess- 
Newsome, and Zeidler (1993) conducted summaries of science education research in 
1989 and 1990, 1991, and 1992, respectively. Science textbook analysis from the three 
summary studies were integrated with recent science textbook studies during 1993 to 
1996. A summary table (Table 1) of the research objectives (data needs), data sources, 
and methodology (instrumentation) suggests a wide number of content analysis 
approaches. 
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Table 1 . 

Summary Chart of Textbooks Content Analysis Research (1989-1996) 



AuthorA" ear 


Data Needs 


Data Source 


Instrumentation 


Anderson & 
Botticelli (1990) 


Investigation in: 

• Quantitative 
techniques to 
measure texts’ 
information 
organization 

• subjective evaluation 
of 12 common 
segments 


• 4 introductory 
biology textbooks 

• 2 analysts 


• numbering relevant 
sentences 

• established rules for 
identifying key words 

• 2 coding schemes 
(explicit & implicit) 

• progression density 
coefficient (contextual 
cue words & new words) 


Barrow (1990) 


• Identify Magnet 
concepts presentation 
styles 

• Identify Potential 
Magnet 

Misconceptions 


10 elementary school 
science textbooks 


• Recording the pages for 
specific concept 

• Composite rating 
instrument (p = prose; 
I=illustration; 
L==laboratory) 


Bazler & Simonis 
(1991) 


Gender equity 
presentation comparison 
between I970’s and 
present editions of texts 


2 versions of each 7 
high school chemistry 
textbooks: 


Replicating a previous study 
in 1970 

• frequency counts with 
1970 study for coding 
scheme comparison 

• inter-rater reliability 


Borger (1990) 


Environmental content 


science & geography 
textbooks & published 
curriculum guidelines 
for 7-12 graders in 
Ontario 


• Environmental 

perspectives framework 


Chiang-Soong & 
Yager (1993) 


Science, Technology, & 
Society topics 


• All pages of the 
1 1 most 

frequently used 
science textbooks 
(Weiss, 1978) 

• 2 raters 


• STS framework from 
project synthesis 

• inter-period rating 
technique (3 weeks) 

• intra-observer 
(consistency .98); inter- 
rater reliability .80 


Chiappetta, 
Fillman, & Sethna 
(I99lb) 


4 Major themes of 

scientific literacy 

• knowledge of science 

• investigative nature 
of science 

• science as a way of 
thinking 

• interaction of science, 
technology, and 
society 


• Randomly 
selected from life, 
earth, physical 
science, 
chemistry, and 
biology textbooks 

• 2 experienced 
raters & 1 raters 
trained by manual 


• adopted & extended 
Garcia’s (1985) 
categories 

• descriptors cards 

• rater’s training manual 

• pilot study to justify the 
framework 

• inter-rater reliability & 
Kappa value for inter- 
rater agreement 
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Author A" ear 


Data Needs 


Data Source 


Instrumentation 


Chiappetta, Sethna, 
& Fillman (1991c) 


4 scientific literacy 
themes & expository 
learning aids 


• 5% of the 5 most 
commonly 
adopted chemistry 
textbooks & 2 
additional 
nationally 
recognized 
chemistry texts 

• 2 raters 


• same procedures as 
previous study 


Chiappetta, Sethna, 
& Fillman (1993) 


4 major scientific literacy 
themes 
& Number of 
vocabularies included 


5% of 5 middle school 
life science textbooks 
& the whole chapters 
specific devoted to 
“What science is?” 
from each textbook 


Same techniques of their 
1991 studies 


deBerg(1989) 


Emergence of 
quantification in pressure- 
volume problems; the 
sequencing from: 

• Learning Theory 
(sequencing, 
emergence, problem 
solving) 

• Nature & method of 
science 


• 28 physics and 
chemistry 
textbooks 

• single rater 


• Theoretical based 
conceptual framework 


deBerg & Treagust 
(1993) 


Gas law sequence. 

Modes of presentation on 
gas properties 

• qualitative modes 

• quantitative modes 


• 14 secondary 
chemistry 
textbooks used in 
Australia 

• 104 teachers 
interviews 


• content coding 
framework 

• teacher questionnaire 


Eltinge & Roberts 
(1993) 


Science as a process of 
inquiry 

• identify key words 

• identify relationship 
between key words 


• 9 editions of high 

school biology 
textbooks Modern 
Biology (1955- 
1988) 


• Linguistic content 
analysis (PLCA - 
computer analytic 
software) 

• logistic regression 


Evans (1989) 


• Discovery/ 
Inquiry/Process 
orientation 

• Hands-on activities 
inclusion 

• Reading & 
vocabulary demands 


• 4“* grade science 
teachers’ manuals 
published by 
Silver Burdett & 
Merrill 

• 1 7 grade 4 
teachers & 360 of 
their students 
were the raters 


• modified CASST 

(Criteria for the Analysis 
and Selection of Science 
Text) 


Fillman (1989) 


4 major scientific literacy 
themes 


• 5% of 1 1 Biology 
textbooks adopted 
by Texas State 

• 2 raters 


• establish the 

appropriateness of 
Chiappetta, et afs 
analyzing manual 
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AuthorAf ear 


Data Needs 


Data Source 


Instrumentation 


Glenn (1990) 


Rosenthal’s (1985) sub- 
topic of evolution: 

• origin of life 

• evidence for 
evolution 

• theories of evolution 

• human evolution 

• evaluation & 
creationism 


• three publishers’ 
high school earth 
science textbooks 
since the 1960- 
1989 


• identify themes 
appearance 

• identify books’ 
preference and 
introduction chapter’s 
presentation styles of 
evolution 

• determine the extent of 
the presentation 

• detailed review the 
extensive coverage 
books but scanning the 
less coverage books 


Hamm & Adams 
(1989) 


Global Problems & 
Issues: 

• population growth 

• world hunger 

• air quality 

• atmospheric issues 

• water resources 

• war technology 


• I0grade6&7 
textbooks 
representing 90% 
used in California 
State 

• 5 raters 


• Issuing survey to 
recognized ‘experts’ in 
science, science 
education, and 
engineering. 

• Analytic framework 
developed based on the 
survey’s findings 

• rating instrument 


Hehr(1989) 


Scientific Literacy 

Preparation: 

• concepts (defined, 
partially defined, 
explained, example) 

• attitudes (present or 
absent) 

• processes skills 
(present or absent) 


• chapters randomly 
selected from 25 
Texas State 
adopted 

textbooks in life, 
earth, physical, 
and biological 
science 

• 5 secondary 
science teachers 
trained as raters 


• Evaluation framework 

• Ground rules 


Holt, Rinehart, & 
Winston 


Science as Inquiry 


• Selected chapters 
(genetics, leaf 
structure, & 
introduction) of 
1956, 1965, 1977, 
& 1985 editions 
of Modern 
Biology 


• Linguistic content 
analysis for categories 

• Regression analysis of 
the categories 


Jeffery & Roach 
(1994) 


Evaluation Protoconcepts 
(topics for preparing 
students study evolution 
in later year) 

• Geological time 

• Natural transition of 
earth environment 

• Variability & 
alteration of genetic 
makeup biotic 
potential 


• 5 elementary & 

middle school 
earth science and 
life science 
textbook series 
(total 7 textbooks) 
by a textbook 
selection criteria 


Conceptual content analysis 
(approaching the data 
“blind”) 

• identify possible data 

• standardized the 
language 

• 17 concepts identified 
through qualitative 
exploring 

• established grounded 
theory 
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AuthorA^ear 


Data Needs 


Data Source 


Instrumentation 


Lin (1990) 


Current Goals of Science 
Education 

• nature of science 

• interrelationship of 
STS 

• openness of 
laboratory activities 

• types of questioning 
style 


• Taiwan junior 
high school earth 
science textbook 


• Framework 

• Intra-rater & inter-rater 


Lloyd (1990) 


Reading comprehension 
of photosynthesis content 

• detailed background 
information 

• analogies 

• examples 


• 3 biology 

textbooks 


• Conceptual framework 
(concepts of the 
requirements, the 
process, the products, & 
the by-products) 

• numbers & nature 


Lumpe & 
Scharmann (1991) 


Contemporary science 
education goals 

• scientific inquiry 

• problem solving 
skills 

• manipulative skills 

• observational skills 


• selective Lab 
activities content 
in Modem 
Biology (1989) & 
BSCS Green 
(1987) 

• 2 raters 


• Content Coding Scheme: 
24 task categories under 
4 major groups (planning 
& design, performance, 
analysis & interpretation, 
and application) 

• Inter-rater reliability .80 


Lumpe & Beck 
(1996) 


Scientific Literacy 
Themes adopted from 
Chiappetta, et. al (1991) 
Descriptive Information 

• no. of chapters 

• no. of pages 

• vocabulary included 


• 7 Biology 
textbooks 5% 
randomly selected 
pages 

• 2 raters (ex- 
biology teachers) 


• Same techniques as 
Chiappetta, et. al. (1991) 

• Inter-rater reliability 


Otero & 

Campanario (1990) 


Characteristics of 
comprehension 


• 6 paragraphs of 
physical science 
passages (4 
contains 
contradiction in 
the second and 
last sentences) 

• secondary science 
students 


• rating 4-point scale 

• short responses on 
problematic sentences 


Selden (1991) 


• Whether Eugenics 
was presented as 
legitimate science 

• Eugenic social 
policies 
recommended 


• 40 high school 

texts 


• texts analysis 

• photographs analysis 
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Author Af ear 


Data Needs 


Data Source 


Instrumentation 


Shepardson & 
Pizzini (1991) 


Expository learning aids 
• cognitive level of 
questions (input, 
processing, output) 


• Stratified 
sampling 
techniques for 
17% of the high 
school science 
texts that 
represent 50% of 
national market 
(total 3 140 
questions were 
analyzed) 

• 2 raters 


• An analysis scheme for 
classification of 
cognitive level of 
questions 


Staver & Bay 
(1989) 


• conceptual structural 

• reasoning demands 


• 1 1 K-3 texts on 

topics of air & 
water 


• conceptual mapping of 
17 segments 


Staver & Lumpe 
(1993) 


The means of 

• introducing Mole 
concept 

• defining Mole 
concept 

• explaining Mole 
concept 


• 29 high school & 

introductory 
college chemistry 
textbooks 


• Cross reading-discussion 

• Reaching agreement 
between raters 


Strube (1989) 


Literary style of 
presentation (readability) 

• Distant authorial 
voice 

• precision 

• context 

• syntax 

• use of rhetorical style 


• physics textbooks 


no specification 


Vachon & Haney 
(1991) 


Level of Abstraction 
(LoA): the ratio of # of 
concepts w/o concrete 
examples to the total U of 
concepts in a text passage 


• 9 samples of 200- 
word passages 
selected from 5, 7, 
10 grades life, 
earth, & physical 
science textbooks. 

• 61 science 
teachers from 
elementary and 
secondary as 
raters 

• 1 1 teachers from 
5, 7, 10 grades 
recognized the 
abstractness & 
students* 
comprehension 


• 24 science educators to 
validate the identifying 
procedures 

• Cronbach’s alpha & 
other criteria to construct 
the LoA score for each 
passage 
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General Research Questions That Require Content Analysis 

A common kind of research uncertainty revealed in the 3 1 content analysis studies 
was; When a science instructional concept (such as scientific literacy, science as inquiry, 
equity in science, etc.) reaches a certain consensus among science educators, researchers 
usually begin to wonder what the status or readiness of science textbooks is, with regard 
to that specific concept -- since science teachers have been found to rely so heavily on 
textbooks. Almost every study cited in Table 1 investigated the weight a specific 
idea/concept carried in science textbooks, from a quantitative perspective. A quantitative 
approach focused on answering question such as “How many pages are or what 
percentage of a science textbook is devoted to scientific literacy?” 

By comparison, a few researchers such as deBerg (1989), Wheeler and Hill 
(1990), and Jeffery and Roach (1994), conducted their studies from a more qualitative 
standpoint. A qualitative approach refers to a study of a specific concept’s presentation 
style such as a mathematically oriented presentation of the pressure-volume concept, or 
the misconceptions produced by diagrams in science textbooks. From the 3 1 studies in 
Table 1 another type of research question, which requires content analysis indicated 
concern with the writing-related issues of science textbooks. For instance, Seldon (1991) 
investigated the use of propositions or analogies as learning aids, and Strube (1989) 
studied the literacy styles of science textbooks as they relate to science learning. 

Common Data Sources for Content Analysis 

Science textbooks and raters who conduct data collection are the data sources of 
textbook content analysis. There are two trends in deciding on the proportion of the text 
to use for analysis. Trend one is studies that randomly selected samples of the texts; five 
studies in Table 1 adopted 5% of the pages from each text. Trend two is a whole-text 
examination for desired content such as: STS inclusion in the whole texts or diagram 
characteristics of every text. Garcia (1985) asserted 5 percent of the pages of each 
textbook’s findings sufficiently large to represent textbook content. Whereas, Hershey 
(1996) warned that the technique of randomly selecting science textbook pages for 
themes analysis may have “easily skewed the results because some chapters emphasize 
[one theme] much more than the other [themes]” (p. 328). The decision of whether to use 
sample pages or whole texts relates closely to the decision of “who will be the raters.” 
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Krippendorff (1980) warned that utilizing a single rater leads to the weakest 
reliability of the resulting content analysis. To assure the reproducibility of any content 
analysis, two or more raters should perform the investigation. In addition, Krippendorff 
alerted researchers who adopt a multiple-rater technique that they have to strictly follow 
the rule of letting raters work independently, even the least communication between the 
framework designer and the raters is not to be allowed. 

Some studies shown in Table 1 were examinations of entire texts, and since 
science textbooks are known for lengthy content, only one rater performing the test-retest 
during a certain period of time is common for a whole-text-analysis. This is because that 
there are too many pages for analysis and there are considerable difficulties in finding a 
skilled rater besides the procedure-developer. One investigator analyzed whole texts in 
the study by Chiang-Soong & Yager (1993), and three weeks later another randomly 
selected textbook had 50 pages randomly chosen from it for a second analysis. The 50 
pages were given to another rater to perform the analysis. This method might provide 
researchers an alternative in reexamining their large amount of data. Chiappetta, Fillman, 
& Sethna (1991a) developed a 25-page training manual for novice raters. Raters 
practiced the technique according to the manual, before starting to analyze the texts 
assigned to them. When raters have comprehensive guidelines establishing how to 
perform content analysis, the problem of finding a skilled rater is minimized. 

Common Instrumentation of Content Analysis 

The methodology applied in these studies varied. The approaches ranged from 
merely counting words, paragraphs, or pages, using a self-defined framework; to simply 
adopting others’ analyzing frameworks and procedures, applied to different data sources; 
or to utilizing thoroughly developed guidebooks incorporating a well-established 
framework through triangulation with “experts” (e.g., science educators, research 
findings, or educational organizations). Constructing conceptual frameworks or 
foundations is only the first step in establishing the validity of content analysis 
(Krippendorff, 1980). However, whether or not the frameworks do measure desired 
content (what they intend to measure), these approaches reflect a more serious need for 
content analysis. One noteworthy example of establishing a valid framework is a study 
by Garcia (1985), who chose scientific literacy themes to orient the research. 
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The frameworks of scientific literacy were developed through several stages. 

First, Garcia identified “descriptors” of scientific literacy by examining relevant literature 
and publications of science education organizations. Each descriptor was put on a card. 
Two science educators were given all the cards to categorize. The procedures generated 
four general but discrete scientific literacy themes, which became the conceptual 
framework. Garcia’s framework was initially targeted at earth science. Chiappetta, et al. 
(1991b) extended the framework to physical and life science and chemistry. This 
framework was applied to five studies included in Table 1 . 

Another approach to obtaining a conceptual framework is oriented toward an 
exploratory style -- a qualitative approach. Adopting qualitative approaches to analyze 
written materials is a common technique used to obtain rich insights and in-depth 
understanding for “un-familiar” concepts (Dey, 1993; Miles & Huberman, 1994). Jeffery 
and Roach (1994) were interested in how “evolutionary protoconcepts” were being 
presented in elementary and middle school science textbooks. Because of the lack of a 
specific list of concepts identified as evolutionary protoconcepts before they started a 
content analysis study, their first task was to collect these concepts. Jeffery and Roach 
adopted a thematic-type approach, in which information (the associated ‘terms’) gleaned 
from the texts is clustered (classified, language standardized, tabulated, and evaluated) 
and presented by the theme (the evolutionary protoconcepts). Generating such a list of 
categories was not a linear process; researchers had to constantly go back to the texts 
simultaneously while analyzing the data in order to validate the categories. Jeffery and 
Roach’s categories were grounded in the data collected. Moreover, their life science 
categories were additionally validated by matching with literature. 

There is no single perfect approach to framework construction. Construction of a 
conceptual framework is closely tied to the nature of the study, and grounded with the 
purposes of the data needs. One rule is that in order to draw reliable inferences from the 
target texts, it is necessary to make valid classifications. A better result is obtained when 
concepts are well-grounded by the literature or validated by “experts.” Thus, the next 
step is to elaborate on the establishment of general, simple, and ready-to-be-used 
procedures. The guidelines can help rater handle data “clerically.” When less judgment 
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is required during applying the conceptual framework, the more likely the technique is to 
be reliable (Krippendorff, 1980). 

A different aspect of instrumentation seen in Table 1 relates to the statistical 
techniques applied. These studies commonly reported reliability of agreement. Inter- 
rater reliability is the agreement coefficient between two raters. Intra-rater reliability is 
the agreement coefficient of two coding results (test and retest). Krippendorff (1980) 
explained the calculation of agreement coefficient, in which maximum agreement is 1 
and 0 means no agreement: 

Reliability is expressed as a fimction of the agreement achieved among 
coders regarding the assignment of units to categories. If agreement 
among coders is perfect for all units, then reliability is assured. If the 
agreement among coders is not better than chance . . . reliability is absent . 

. . reliability . . . always boils down to measuring the agreement achieved 
among observers, coders, or judges regarding how they independently 
process scientific information, (p. 133) 

The agreement coefficient (a) is derived from a process considering percentages of 
“agreement by chance” (the agreement presumably occurred merely by chance), 
“observed co-occurrences” (the matched data set occurred between raters), and 
“maximum agreement” (the calculated maximum matched incidence rate occurred 
between raters). In defending the actual or “absolute” agreement value (the observed co- 
occurrence) instead of Krippendorff s agreement coefficient, Tinsley and Weiss (1975) 
suggested that the actual inter-rater agreement is more appropriate when the data is 
continuous (nominal scale). The agreement coefficient a is more useful for categorical 
data. However, Krippendorff (1980) asserted that the agreement derived from different 
raters always has the possibility of occurring by chance. “When the coefficient a is only 
10% . . . conclusions to which such data would lead are largely misleading or true only by 
chance. The fact that 60% . . . matched, turned out to have no meaning” (p.l35). 

For nominal data, Fillman (1989) introduced Cohen’s (1960) kappa because it 
takes agreement by chance into consideration. As reported by Chiappetta, et al. (1991b), 
the kappa statistic is applied to compute inter-rater agreement when (a) two raters are 
working independently, (b) the units of analysis are independent, and (c) the categories 
are independent, mutually exclusive, and contain nominal data. Krippendorff (1980) 



O 

ERIC 



14 16 



Science Textbooks Studies Reanalysis 



believed that there is no set answer for “how high the agreement is preferred?” He 
implied that the agreement coefficient is more context-grounded. Cohen’s (1960) kappa, 
on the other hand, was agreed upon by Rubenstein and Brown (1984), which kappa 
greater than .75 indicates excellent agreement beyond chance; kappa between .40 and .75 
represents fair to good agreement beyond chance; kappa below .40 indicates fair to poor 
agreement beyond chance. 

All researchers in the Table 1 reported that a high reliability obtained. Despite the 
fact that most studies were not clear about data-type collected, the absolute agreement 
percentages were cited as evidence of “reliability.” These studies are wholly deceptive in 
the ill-methods. Five studies in Table 1 reported kappa values to sustain their agreement 
coefficient. The study by Chiang-Soong & Yager (1993) included a disagreement 
percentage (disagreement among both sets of data) in their study to filter out “unreliable” 
data. 

Data reliability is another feature explained by Krippendorff (1980) as one 
important statistical technique to be considered in content analysis. “The ultimate aim of 
testing reliability is to establish whether data obtained in the course of research can 
provide a trustworthy basis for drawing inferences, making recommendations, supporting 
decisions, or accepting something as fact” (p.l46). There is no single study in Table 1 
addressing data reliability. 

Finally, Krippendorff (1980) urged that many preliminary and detailed reliability 
tests be conducted during the framework development stage and before the data 
reliability tests. This provides early warning of the potentially unreliable sources. It may 
lead to modifications of the analytic categories, changes in coding instructions, or 
decisions on choosing qualified raters. 



Conclusions 

Table 1 summarizes studies that applied content analysis to review textbooks by 
science educators published during 1989 to 1996. The four categories: research 
questions, research objectives (data needs), data sources, and methodology 
(instrumentation) has provided a database to reflect several issues such as reliability and 
validity in qualitative data analysis indicated by Miles and Huberman (1993). It also 
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presented a few common strategies likely to be adopted when researchers conducted a 
textbook analysis. The reanalysis of past science textbook studies has suggested a wide 
numbers of content analysis approaches applied, and yet, there was no consensus in the 
criteria used to determine the appropriateness of such research methodology. This study 
found that most of the researchers had fragmented uses in content analysis as a textbook 
study method. Trends found in the reanalysis have provided a basis for inquiry of 
approach likely to be adopted by researchers. 

With such information, questions arose for how these studies can be utilized by 
school science teachers. A pilot Textbook Decisions Questiormaires was designed and 
administrated by fifty-six science teachers at the Los Angeles Unified School District. 
The reliability (a) was not higher than .60, which indicated the need for revision. 
According to TIMSS population one (3^** and 4^** grader) survey to school policy decision 
of determining textbooks, out of 189 schools sampled, there are 69 schools (36.5%) 
reports that teachers have greater responsibility in making the decision. Thus, there is a 
pressing need: to fiirther provided teachers with proper tools and skills that help them to 
make decision through study like this. 

One crucial lesson learned from this pilot survey reflected TIMSS report - 
teachers tend to try to “use” all topics displayed in science textbooks (Schmidt, 
McKnight, & Raizen, 1997). Most teachers believed that the decisions of whether to 
adopt a science textbook or not is related to their future use of the book to guiding their 
instruction. Teachers also indicated that the when inspecting a science textbook, most of 
them tend to choose the section/topic interested them to examine. They tended to have 
limited time to make their final decision. 

Lastly, while science textbooks still play a profound role in science instruction 
and learning, this study explored the need for generating an effective and efficient 
approach or “tool” to conduct examination in textbooks for science teachers. The gap 
between the approaches by researchers and teachers is wide. In order to design a teacher 
“friendly” content analysis method, researcher must start to survey teacher’s needs. 
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